Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Analysis of Temporal-Difference Learning

We present new results about the temporal-difference learning algorithm, as applied to approximating the cost-to-go function of a Markov chain using linear function approximators. The algorithm we analyze performs on-line updating of a parameter vector during a single endless trajectory of an aperiodic irreducible finite state Markov chain. Results include convergence (with probability 1), a ch...

Instance Optimal Learning

We consider the following basic learning task: given independent draws from an unknowndistribution over a discrete support, output an approximation of the distribution that is as ac-curate as possible in `1 distance (equivalently, total variation distance, or “statistical distance”).Perhaps surprisingly, it is often possible to “de-noise” the empirical distribution of the samples<lb...

An Analysis of Experience Replay in Temporal Difference Learning

An Analysis of Temporal-Difference Learning with Function Approximation

We discuss the temporal-difference learning algorithm, as applied to approximating the cost-to-go function of an infinite-horizon discounted Markov chain. The algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic Markov chain with a finite or infinite state space. We present a proof of convergence (with pr...

Bayes Optimal Instance-Based Learning

In this paper we present a probabilistic formalization of the instance-based learning approach. In our Bayesian framework, moving from the construction of an explicit hypothesis to a data-driven instance-based learning approach, is equivalent to averaging over all the (possibly innnitely many) individual models. The general Bayesian instance-based learning framework described in this paper can ...